…ignature

HierAllGather now auto-detects the node-local rank count (LOCAL_WORLD_SIZE, else
hostname grouping, else npes) so callers use the same constructor/call signature
as the flat AllgatherSdma with no new required argument. ranks_per_node is now
optional and keyword-only; added transit_buffer_size for signature parity.
Single node still degenerates to the pure intra-node SDMA path.

Resolve conflicts with the upstream param-contiguous SDMA allgather:
- oneshot_sdma_kernel.hpp: keep both the hierarchical sub-group/broadcast SDMA
  kernels and the upstream param-contiguous kernel (additive).
- symmetric_memory.cpp: keep deviceHandles_d indexing by global pe (the array is
  worldSize-sized and all SDMA kernels index by global pe); adopt the upstream
  non-fatal GPU-metadata teardown.

Add HierAllGather.all_gather(tensor_list, tensor) matching
torch.distributed.all_gather (list output), built on the same hierarchical
intra-node SDMA / inter-node RDMA path as the contiguous all_gather_into_tensor.
Bit-exact vs torch across {bf16,fp16,fp32,int32}; adds test_hier_allgather_list.

…erlap bench)

…DMA + inter-RDMA)

…A + inter RDMA)

Adds a single drop-in FSDP2 AllGather backend, MoriAllGather, used identically
for single-node and cross-node runs via the stock
FSDPModule.set_custom_all_gather API. It routes intra-node traffic over SDMA
copy engines (XGMI) and, when the process group spans multiple nodes, inter-node
traffic over RDMA — the same object handles both, so user code is unchanged
between one node and many. (MoriHierAllGather kept as a backward-compat alias.)

Includes the Qwen-7B FSDP2 step benchmark, a 2-node driver, and the chart
script. No mori source change needed; the HierAllGather primitive already exists
and handles the single-node case as pure intra-node SDMA.

…l N*P launch overhead to beat RCCL xnode)

Motivation: zero-copy killed FSDP copy-out but the per-(node-block,param) loop
issued N_nodes*N_params SubGroup launches per all-gather; that launch overhead
regressed 2-node FSDP to 106 TFLOPS (< RCCL 128). New fused
OneShotAllGatherSdmaSubGroupParamContiguousKernel loops all blocks+splits inside
one launch. 2-node standalone bit-exact PASS bf16/fp16/fp32/int32.

…the inter-node RDMA ring (recover ring||gather overlap zero-copy lost to serial path; lever to beat RCCL xnode)

…fault under FSDP; standalone bit-exact)

The ring||local-scatter overlap zero-copy path is bit-exact in the 2-node
standalone test but triggers an HSA memory-exception under FSDP's repeated-call/
buffer-reuse pattern. Keep the proven non-overlap fused scatter as the default
zero-copy path; enable overlap with MORI_HIER_PC_OVERLAP=1 to iterate the fault.

…ng path); catches non-bit-exact scatter (num_blocks=1/W=G) behind the +17.6% HSDP result

… register-once regression test (isolates num_blocks=1 scatter bug; num_blocks=N cross-node path re-confirmed bit-exact)

…ntiguous kernel (align to proven flat kernel); add receiver-slot diagnostic to intra test

Fused subgroup param-contiguous scatter proven buggy via single-node repro
(world=4 G=4 num_nodes=1): concurrent-warp multi-put scrambles/drops sender
data despite correct offsets. threadfence_system (SDMA async, unaffected) only
shifted the pattern. Per-put gather_kernel_direct loop predecessor was bit-exact.

…per-op cross-node barrier is the FSDP gap (SDMA 102.9->112.1, beats RCCL 105.6 when removed); justifies generation-counter barrier-free ring

…item()/.tolist() drained the pipeline every call, destroying AG<->backward overlap under FSDP); cache u32 split tensors

…intra barriers) — decisive A/B falsifies barrier-skew: all-barrier removal gives 0% recovery (111.98->111.79), gap is not per-op barrier serialization

… was an undersized-output IPC-registration artifact, NOT a concurrent-put race

Root cause (proven via fast single-node nproc=4 repro + a size sweep): the intra
param-contiguous direct scatter writes to peerPtrs[remotePe]+dstBaseOffset assuming
the registered peer pointer == the output buffer base. When the standalone test's
output was small (bf16, ~16MB) torch SUB-allocated it inside a larger pool segment,
so ShmemSymmetricRegister/hipIpcGetMemHandle resolved the peer pointer to the SEGMENT
base (not the buffer) and the scatter SILENTLY CORRUPTED (wrong slots). float32
(2x larger) crossed the own-segment threshold and passed, which made the failure look
dtype/num_blocks-specific. Enlarging the output to its own segment makes num_blocks=1
bit-exact for bf16/fp16/fp32/int32 (12 reps). The kernel was never buggy; a race is not
dtype-size-deterministic. Bumps _PARAM_SPLITS to force own-segment; adds probe_nb.py.

…contiguous scatters (shared per-groupPos flag slot)

The overlapped param-contiguous zero-copy path (MORI_HIER_PC_OVERLAP=1, the historic
best 113.9 TFLOPS win candidate) ran the side-stream LOCAL-block scatter concurrently
with the main-stream REMOTE-block scatters. Both call gather_kernel_direct_param_contiguous,
which shares ONE per-groupPos flag slot + seq token on the intra handle; concurrent use
let a receiver observe the other scatter's flag bump -> premature completion / spin-deadlock
under FSDP (the Turn 4/14 hang). Move main.wait_stream(side) BEFORE the remote loop so the
two scatter phases are serialized. The key overlap (side local scatter || inter-node RDMA
ring) is preserved: the ring finish is enqueued on main before the wait, so it still runs
concurrent with the side scatter; only the ring-dependent remote scatters wait. Kernel
correctness itself is now proven bit-exact (see prior commit).

…ut on the side stream so the caching allocator does not recycle FSDP buffers mid-scatter

…hurn per AG call (each change = a cross-node ShmemSymmetric register/deregister collective that cannot overlap; candidate for the in-FSDP per-AG inflation vs RCCL)

…oss-nondeterminism is an async completion-capture bug — forced stream.synchronize() at op return gives bit-exact loss==native (11.0992556 x2), while on-device ShmemBarrierOnStream does not drain the local SDMA copy-engine DMA before FSDP consumes the output

…ns on the peer-completion flag with a SYSTEM-scope acquire (AtomicLoadSeqCstSystem) + __threadfence_system instead of an AGENT-scope relaxed load; the flag+data are produced by a REMOTE peer GPU (different HSA agent) so AGENT-scope gives no cross-agent happens-before and the copy-OUT could consume not-yet-visible data under rapid FSDP reuse (host sync masked it)

… elementwise copy-out) to test the copy-engine<->CU coherence root cause for FSDP loss-drift; both CU-write-output and CU-read-transit variants leave loss in the same drift band => the stale bytes are in out_ itself (SDMA receiver drain), not the copy-out engine

…iguous zero-copy — reproduces the FSDP AG->backward hazard (AG on comm stream, consumer waits on a recorded event on the main stream, per-rep varying inputs + layout-weighted consumer + compute pressure); proves the zero-copy direct path is bit-exact AND deterministic under overlap at ~2M-elem splits => the FSDP loss drift is NOT in the AG kernel completion at this regime (look to copy-out path / small-layer routing next)

… deployed FSDP perf path) + small/single-split size profiles — cross-node world=8 shows BOTH the copy-out and zero-copy AllGather paths are bit-exact AND deterministic under the FSDP AG->consumer cross-stream overlap at every size band (0/40 wrong); NaN-safe verdict. => the ~0.15% FSDP loss drift is NOT in HierAllGather completion ordering; redirect to the downstream FSDP path (reduce-scatter / bf16 accum) or an internal side-stream event-join not captured by the caller event

… mode; mixing copy-OUT __call__ and zero-copy enqueue_param_contiguous on ONE handle contaminated shared intra flag/seq + output-registration state (copy-OUT registers transit out_, zero-copy registers the USER output), spuriously yielding a stable-NaN for [zerocopy bf16 nsplit=5] right after copy-OUT ops. The pure-mode bit-exact test PASSES bf16 at that exact config, and FSDP uses one mode per run, so per-mode handles are the faithful harness. Now 12/12 configs 0/40-wrong => both AG output paths bit-exact + deterministic under FSDP AG->consumer cross-stream overlap at every size band; the ~0.15%% FSDP loss drift is NOT in the AG

…ther truth ref

Large profile (68M elems/rank; gathered int32 ~2.18GB, crosses 2^31 bytes) probes
u32 byte-offset overflow at the embed+lm_head band -> 2-node bit-exact PASS
(bf16/fp32/int32), ruling out size-dependent AG corruption. Overlap UT now checks
the async output against an INDEPENDENT all_gather reference (not the self-golden),
so a stable-but-wrong copy-out drain would be caught -> still 0 wrong at all bands,
falsifying the in-kernel drain-race theory for the FSDP loss drift.

… receiver flag wait

The InterNodeRing receiver spun on the completion flag with core::AtomicLoadRelaxed
and no acquire fence, unlike the intra SDMA gather (AtomicLoadSeqCstSystem +
__threadfence_system). The flag is bumped by a REMOTE peer's RDMA AMO and the chunk
it guards is landed by that peer's RDMA put -- both cross-agent writes; a relaxed load
establishes no happens-before, so the received data need not be coherently visible to
the forward-put / copy-out. Harden to a system-scope acquire matching the proven intra
pattern. NOTE: in-situ AG-output probe (MORI_FSDP_AG_VERIFY) shows this alone does NOT
close the FSDP loss race (184/384 AG calls still read stale vs RCCL, 0/384 under host
sync) -- the residual race is in the copy-out drain (ring-buffer->output visibility), the
next target.

af86cb1 over-reached and deleted pre-existing all2all/allreduce/
allreduce_async/rccl_allgather tests unrelated to this feature. Bring them back.

Restore anvil.cpp / shmem_ibgda_kernels.hpp / init.cpp to main except the
functional bits (anvil HIP-ordinal->HSA-agent BDF fix), and drop stray comment/
whitespace rewords in symmetric_memory.cpp + mlx5.cpp. Keeps the PR diff to
feature-relevant changes only.

Both were set True in __init__ with no ctor arg and no env override, so
every else arm behind them was dead. collective.py still defaults
stream_ring=False, so the call sites now pass True literally.

Ctor-only opt-ins, both mutually exclusive with slice_inter which defaults
True, so the ctor raised ValueError for any combination that would reach
them. Nothing in the repo passes either kwarg.

442 lines with a three-way path switch in one method. Pure extraction into
_call_pipe_band / _call_sliced / _call_sliced_fused_remote /
_call_sliced_fused_local / _call_nonsliced; _call_impl is down to the
dispatch. Stream, event and barrier order is unchanged.

Last references keeping those two branches reachable. The three
fuse_barrier scenarios stay -- that knob is live on the shipped path.

Stored and never read; copy-paste from _HostProxyDeferredWork where it is
live. Dropping the ref is safe because bench.py clamps prefetch depth to 1,
so wait() always precedes the next __call__ -- _get_collective can rebuild
the collective on a cap change, so the Work object's reference is not
guaranteed to be a non-last one.

Both files already had their own copyright line before the license hook
prepended the canonical block.

The direct path needs num_nodes >= 2 over RDMA. On a single node the file
printed SKIP, asserted nothing and still reported PASSED.

leader_only / gather_in_place / out_in_place are gone from the product
code, so the stub no longer has to define them.

- index the SDMA handle arrays by a clearly-named global pe slot
- de-duplicate the within-node device-id computation into one helper
- match a HIP device to its HSA agent only on a unique BDF, keeping the
  identity fallback when the low 16-bit BDF is ambiguous

Resolve the SDMA-channel-cap constant duplication in favor of upstream #501's
kMaxSdmaChannelsPerPair (drop the local kSdmaMaxNumChannels alias).

…pile

--sizes-mb '8 64' lost its quotes when node2's argv went through
ssh "${NODE2_CMD[*]}" (a second word-split), so '64' hit the wrapper as a
bare arg -> 'Unknown option: 64', node2 died, node1 hung to timeout (exit 124).
use comma-separated 8,64 and split it back inside the wrapper.

RCCL's cross-node RoCE bring-up was failing on the internode runners
(MI355/AINIC: ibv_modify_qp timeout, GID index 0 N/A; MI325/BNXT: QP local
access violation) -- and that happened in the RCCL reference collective, before
mori's own AllGather ran, so the smoke test never actually exercised mori.

compute the expected output locally instead (rank-major concat of each rank's
deterministic input) and drop device_id= from init_process_group so NCCL stays
lazy and never brings up RoCE QPs. now the only cross-node path is mori's.
verified bit-exact on 2-node MI300X+mlx5 with no NCCL_IB_* set.